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EXECUTIVE SUMMARY 



Many studies have documented the fact that college graduates earn significantly more 
money during their lifetimes than high school graduates (U.S. Department of Education 2004). 
Research also shows that college graduates demonstrate greater civic involvement and are more 
likely to vote and assume leadership roles in their communities (Astin 1993; Bowen and 
Bok 1998). 

Low-income students and students whose parents have not attended college typically are 
less likely than middle- and upper-income students to complete high school and attend college, 
and are thus less likely to reap the benefits of attending college. Lack of information, resources, 
and exposure to others who have navigated the college process may be substantial hurdles for 
these students. Lederal financial aid is available through Pell Grants, college tuition tax credits, 
and student loan programs, but low-income students may not be taking full advantage of these 
sources. Even low-income students with high educational aspirations may find the financial aid 
and college application processes overwhelming and discouraging. 

In 1965, Congress established the Talent Search Program as one of the original federal 
TRIO programs. The others include Upward Bound, created in 1964, and Special Services (later 
renamed Student Support Services), established in 1968. Today, TRIO includes five other 
programs. Collectively, these programs help low-income, potentially first-generation college 
students prepare for and gain access to college. The Talent Search program primarily provides 
information on the types of high school courses students should take to prepare for college and 
on the financial aid available to pay for college. The program also helps students access 
financial aid through applications for grants, loans, and scholarships, and orients students to 
different types of colleges and the college application process. In fiscal year (LY) 2004, the 
Talent Search program received approximately $144 million to serve 382,500 students in 
470 projects across the country; on average, each project spent approximately $375 per 
participant served. The program has grown since LY 2000, the period relevant to this study, 
when Talent Search served 320,000 students and spent approximately $313 per student. 

After a two-year implementation study, the U.S. Department of Education’s Policy and 
Program Studies Service selected Mathematica Policy Research Inc. (MPR) in 2000 to assess the 
effect of Talent Search in selected states. A variety of designs were considered and ultimately 
the study team opted to compile data from administrative records from many sources, including 
program, state, and federal records, to evaluate the effectiveness of federal education programs, 
partly as a test of whether such an evaluation was feasible. The study also yielded useful 
information about the effectiveness of the Talent Search program. 



METHODOLOGY 

The study included an analysis of the effectiveness of the Talent Search program in Llorida, 
Indiana, and Texas. We based our analysis on administrative data compiled in these three states 
and a quasi-experimental design to create matched comparison groups (Rosenbaum and Rubin 
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1985). Outcomes of students who participated in Talent Search were compared with outcomes 
of similar students at the same schools or other schools who did not participate. We restricted 
our analysis to the cohort of students who were in ninth grade in 1995-96 to allow us to collect 
information on high school completion and postsecondary enrollment, which occurred as late as 
2002. Although all students were in ninth grade in 1995-96, Talent Search participants may 
have received services through the program at any point from grades six through twelve. We 
compared secondary and postsecondary outcomes between Talent Search participants and 
comparison groups within each state. 

We received data on Talent Search participants from at least 60 percent of all Talent Search 
programs operating in Florida, Indiana, and Texas in 1995-96. As shown in Table 1, samples 
included large numbers of Talent Search participants and matched comparison students in 
each state. 



TABLE 1 

NUMBER OE TAEENT SEARCH PROJECTS, TAEENT SEARCH PARTICIPANTS, 
AND COMPARISON STUDENTS, BY STATE 



State 


Total Number of 
Talent Search 
Projects Operating 
in 1995-96 


Number of Projects 
Providing Data 


Number of Talent 
Search Participants 
in Study Cohort 


Number of Matched 
Comparison 
Students in Study 
Cohort 


Elorida 


7 


5 


908 


13,843 


Indiana 


8 


7 


1,166 


9,844 


Texas 


16 


10 


4,112 


30,842 



KEY FINDINGS 

The main research questions addressed in this report are as follows. 

1. Is it possible to rely on administrative records to compile a retrospective record of 
participation in Talent Search, characteristics of students in secondary school, and 
secondary and postsecondary outcomes? 

2. Can administrative data and quasi-experimental techniques be used to identify 
students who are not participating in the program but have characteristics similar to 
the Talent Search students in order to estimate the relationships between 
participation in Talent Search and secondary and postsecondary outcomes? 
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The Efficacy of Using Large State Databases to Inform Policy 



The compilation of data from administrative sources to study the effect of Talent Search on 
participants was successful in Florida, Indiana, and Texas. We could not compile a suitable data 
file for analysis in Minnesota (due to a lack of access to state secondary school records) and 
Washington (due to missing or poor-quality Talent Search project data). Obtaining student level 
data which included information identifying students to facilitate merging records across data 
sources was challenging to obtain for the years of interest, 1995-2000. Data from recent years 
should be easier to attain as more states develop systems for compiling secondary and 
postsecondary school records, and federal programs are more consistent in reporting information 
on the participants served and maintain records electronically. 

The data files compiled in Florida, Indiana, and Texas contained a wealth of information on 
students in Talent Search. This included important demographic information such as age, race, 
and gender; the school the student was enrolled in for ninth grade; and postsecondary outcomes, 
such as first-time application for financial aid and postsecondary enrollment. Because we 
compiled a large amount of data in each state, both in terms of the number of data elements 
available and the size of the student samples, we were able to use complex propensity score 
matching models to identify nonparticipating students who were most similar to Talent Search 
participants. We were also able to obtain these data at a far lower cost and in a shorter time 
frame than if we had collected data directly from students and their families over a five- to 
seven-year period. 

However, not all data on student characteristics and on secondary or postsecondary 
outcomes of interest were available in any one state, and the type of information that was 
missing differed across states. However, the variations in the data, as well as our strategies for 
drawing comparison students in each state, provided a good test of the sensitivity of the findings 
to using different data sources and comparison students. Findings that were consistent across all 
three states suggest greater robustness. 



Findings Across States 

Talent Search participants were more likely than comparison students to apply for federal 
financial aid and enroll in public postsecondary institutions in Florida, Indiana, and Texas. 

• Financial Aid Applications. Talent Search participants were more likely than 
nonparticipants from similar backgrounds to be first-time applicants for financial aid 
in the 1999-2000 school year. The difference in financial aid application for Talent 
Search participants and nonparticipants was 17, 14, and 28 percentage points, 
respectively, for Florida, Indiana, and Texas (Figure I). The difference was smallest 
in Indiana, where we had the strongest measures of educational aspirations. Even in 
Indiana, however, the gap represents application levels for financial aid that are one- 
third higher for Talent Search participants. 
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FIGURE 1 



TALENT SEARCH PARTICIPANTS AND COMPARISON STUDENTS WHO WERE 
EIRST-TIME APPLICANTS EOR EEDERAL EINANCIAL AID, 1999-2000, 

BY STATE 




■ Talent Search Participants □ Comparison Students 

Note: Talent Search participants were students in ninth grade in 1995-96 served hy a Talent Search project 
that supplied data for this study. Comparison students are nonparticipants from the same target high 
schools as Talent Search participants who matched to a Talent Search participant. 

***Difference is statistically significant at the 1 percent level. 



• Postsecondary Enrollment. Talent Search participants were more likely than 
nonparticipants to enroll in a public college or university in their state by the 1999- 
2000 school year. Initial enrollment in a postsecondary institution was higher by 14, 
6, and 18 percentage points, respectively, for Florida, Indiana, and Texas (Figure 2). 

• Enrollment in Two- Versus Four-Year Institutions. Talent Search participants were 
more likely to enroll in two-year and four-year institutions and the gains were larger 
and more statistically robust for two-year enrollment (Figure 3). Enrollment by type 
of institution (two- or four-year) was linked to the type of institution hosting the 
Talent Search project. In general, projects increased two- or four-year enrollment, 
but not both. Talent Search projects may have increased enrollment by exposure to 
their type of institution, or to their specific institution. 
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FIGURE 2 



TALENT SEARCH PARTICIPANTS AND COMPARISON STUDENTS WHO 
ENROLLED IN A PUBLIC POSTSECONDARY INSTITUTION, 1999-2000, 

BY STATE 




■ Talent Search Participants □ Comparison Students 

Note: Talent Search participants were students in ninth grade in 1995-96 served hy a Talent Search project 
that supplied data for this study. Comparison students are nonparticipants from the same target high 
schools as Talent Search participants who matched to a Talent Search participant. 

***Difference is statistically significant at the 1 percent level. 



We also examined differenees in high sehool eompletion in Florida and Texas, the states 
where this information was available in administrative reeords. There were large differenees in 
Florida and Texas — 9 and 14 percentage points, respectively — between Talent Search 
participants and nonparticipants. However, we are less confident this finding can be attributed to 
participation in Talent Search than the findings on application for financial aid and initial college 
enrollment. Estimates of the differences in Florida and Texas could be biased upward if Talent 
Search project staff chose to serve students who were most likely to complete high school and 
most Talent Search project services do not directly target high school retention, so there is no 
program-based explanation for generating a difference of this magnitude. Also, we cannot verify 
that Talent Search participants also had higher rates of high school completion in Indiana. The 
magnitude of the differences in high school completion suggests that the data available in Florida 
and Texas may not be adequate to control for characteristics, such as educational aspirations and 
motivation, of students who choose to participate in Talent Search. It is possible that some of the 
difference in high school completion is due to participation in Talent Search, but the evidence is 
not as strong as the findings on application for financial aid and postsecondary enrollment. 
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FIGURE 3 



TALENT SEARCH PARTICIPANTS AND COMPARISON STUDENTS WHO ENROLLED 
IN A PUBLIC TWO-YEAR OR EOUR-YEAR INSTITUTION, 1999-2000, 

BY STATE 



Two-Year Institution 



Eour-Year Institution 




Florida*** Indiana*** Texas*** Florida*** Indiana Texas*** 



■ Talent Search Participants □ Comparison Students 



Note: Talent Search participants were students in ninth grade in 1995-96 served hy a Talent Search project 
that supplied data for this study. Comparison students are nonparticipants from the same target high 
schools as Talent Search participants who matched to a Talent Search participant. 

***Difference is statistically significant at the 1 percent level. 



CONCLUSION 

The study was not a randomized experiment and we cannot attribute differences in outcomes 
solely to participation in Talent Search. However, despite the limitations in the data and the 
design, we are more confident that some of the differences in first-time application for financial 
aid and initial postsecondary enrollment can be attributed to participation in Talent Search, for 
several reasons. First, the magnitude of the differences in first-time application for financial aid 
and postsecondary enrollment were larger than the difference in high school completion and 
remained even when examining these outcomes among high school graduates only. This 
suggests that even if unmeasured characteristics accounted for all the differences in high school 
completion, some of the differences in postsecondary outcomes could still be attributed to 
participation in Talent Search. Second, the higher rates of application for financial aid and 
postsecondary enrollment were consistent across all three states (although the differences were 
smallest in Indiana, where the two groups had similar educational aspirations). 
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A third and important reason is consistency with the program’s objectives and services it 
delivers. Talent Search projects report directly targeting services toward activities to increase 
application for financial aid and postsecondary enrollment. From the research conducted for 
Phase I of this study, we found that help with application for financial aid is “hands on” — it 
involves sitting with students to complete their paper- and-pencil applications or at a computer to 
complete the Web-based Free Application for Federal Student Aid (FAFSA). Talent Search 
project staff also reported providing college orientation activities, such as help completing 
college applications, visits to college campuses, and arranging for participants to shadow 
students — often Talent Search alumni — at the project’s host institution. Looking at the findings 
across projects, the potential relationship between enrollment in a two- or four-year institution 
and the type of institution hosting the Talent Search project suggests that orientation to the host 
institution itself may increase postsecondary enrollment for participants. 

The findings we present in this report suggest that assisting low-income students who have 
college aspirations to overcome information barriers — an important objective of the Talent 
Search program — may be effective in helping these students achieve their aspirations. Practical 
information — direct guidance on how to complete applications for financial aid and admission to 
college and what a college campus looks and feels like — may have been one of the key services 
that Talent Search projects delivered. 
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I. INTRODUCTION 



A. PROGRAM DESCRIPTION 

One of the original federal TRIO programs, Talent Search was designed to help low-income, 
potentially first-generation college students prepare for and gain access to college. The program 
helps students achieve these goals by: 

• Providing information on the types of high school courses students should take to 
prepare for college 

• Providing information on financial aid available to pay for college 

• Helping students access financial aid through applications for grants, loans, 
and scholarships 

• Orienting students to different types of colleges and the college application process 

Talent Search has consistently been the largest of the TRIO programs in the number of 
students served. Talent Search funding in fiscal year (FY) 2004 was approximately $144 million 
and it served 382,500 students in 470 projects across the country. The program has grown since 
FY 2000, the period relevant to this study, when Talent Search served approximately 320,000 
students in 360 projects across the country.^ Talent Search, which spent approximately $313 per 
student in FY 2000, is a relatively low-intensity program. It targets students who have 
academic potential for college but who may need extra information and help in navigating the 
financial aid and college application process. The low-intensity structure of Talent Search is 
reflected in the limited number of services participants received. According to a survey of 

• Other original TRIO programs were much smaller in the same fiscal year. In 2000, Student Support Services 
(SSS) served approximately 180,000 students. In 1999, Upward Bound served fewer than 60,000 students 
combined. For a profile of TRIO programs, see U.S. Department of Education 2003. 

^ For example, in FY 2001, the average cost per student served by Upward Bound was $4,800 per year. 
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project directors in 2000, while almost three of ten Talent Search participants received 20 or 
more hours per year of services, nearly half received 10 hours per year of services or fewer.^ 

1. Context for Studying Talent Search 

The importance for success in the labor market of completing high school and earning a 
postsecondary degree is well documented. The difference in earnings between high school and 
college graduates is great, even among young workers. For full-time workers ages 25 to 34, 
median earnings are 65 to 70 percent higher for workers with a bachelor’s degree than for 
workers with a high school diploma only. Workers who do not complete high school fare much 
worse, earning only 75 percent of what high school graduates earn and less than half of what 
workers with a bachelor’s degree earn (U.S. Department of Education 2004). These differences 
increase with workers’ ages, as earnings also rise more rapidly among college-educated workers 
with experience (Murphy and Welch 1992). In addition to the economic benefits of a college 
education, evidence suggests positive externalities are generated by individuals who complete 
college, such as having a greater civic orientation, being more likely to vote, and assuming 
leadership roles in their communities (Astin 1993; Bowen and Bok 1998). 

Low-income students are less likely than middle- and upper-income ones to complete high 
school and attend college (and thus less likely to reap the benefits of attending college) for 
several reasons. First, the barriers low-income students face are due, in part, to the 
socioeconomic status of their families — it is more difficult for low-income families to allocate 
time and money toward schooling and away from other necessities. Second, the environment 
outside the household compounds these issues. Recent studies of high school dropout rates 
across the country indicate that not only do low-income students have higher dropout rates than 

^ For details on the program as it was operating in the late 1990s, see Cahalan et al. 2004. 
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other students, but middle-income students at high-poverty high schools have higher dropout 
rates than middle-income students at other high schools. Attending a high-poverty high school is 
particularly deleterious for high school completion among African Americans (Swanson 2004; 
Balfanz and Legters 2004). Third, most low-income students do not have a parent who has a 
college degree, which presents both financial and informational barriers for students accessing 
financial aid and navigating the college admissions process. In 1999, 82 percent of students with 
parents with a bachelor’s degree enrolled in college immediately after high school, compared to 
54 percent of students with parents with high school diplomas only. Differences in high school 
curricula, parents’ education, and test scores can partly explain the gap in enrollment rates 
between low- and high-income students (U.S. Department of Education 2001; Jacobson 
etal. 2001; McPherson and Schapiro 1998). However, evidence indicates that, even though 
financial aid programs such as Pell Grants, college tuition tax credits, and student loan programs 
are available, differences in financial resources also play an important role (Kane 1999). Talent 
Search tries to help students overcome these obstacles by primarily targeting low-income, 
potentially first-generation college students."^ 

Additionally, low-income students are not taking full advantage of financial aid programs. 
A study by the American Council on Education showed that 20 to 30 percent of college-going 
students with family incomes below $40,000 (and thus likely to be eligible for federal financial 
aid) did not file a Eree Application for Eederal Student Aid (EAESA) in 2000. This includes 
approximately 850,000 students who may have been eligible for a Pell Grant (King 2004). Other 
studies have also shown that while low-income students may have high educational aspirations. 



In each Talent Search project, up to one-third of participants need not meet either of these criteria. Most 
projects, however, have considerably fewer than one-third of participants who do not meet either of the criteria. 
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they find the college admissions test-taking, financial aid application, and college application 
process difficult to navigate (St. John et al. 2002; Avery and Kane 2004). 

2. Services Offered by Talent Search 

Talent Search helps students overcome the informational barriers that stem from attending 
high-poverty high schools and being the first generation in their families to attend college. The 
services and goals of the Talent Search program are specifically intended to address some of the 
barriers described above. Talent Search projects reported providing a wide range of services in 
1999-2000, but the services most commonly provided fall into three categories: (1) academic 
support, (2) career development, and (3) financial aid. Table I.l shows the percent of projects 
providing the services most commonly reported in those categories. The services listed include 
only those that more than 90 percent of the projects provided. In addition, project directors listed 
financial aid services and visiting college campuses as the top two services that contributed to 
achieving the projects’ objectives. 

3. Study Description and Report Overview 

Assessing the effectiveness of Talent Search in achieving its program goals is challenging, 
due to the large number of students it serves, the diffusion of projects across all 50 states and 
many schools, and the low intensity of services. Because many services focus on providing 
information to students, conducting an evaluation using a randomized design or nonrandomized 
study based on a longitudinal survey of students could be difficult to implement and 
prohibitively expensive. Based on three designs proposed by Mathematica Policy Research Inc. 
(MPR) in 2000 (Maxfield et al. 2000), the U.S. Department of Education (ED) chose to study the 
effect of participation in Talent Search on secondary and postsecondary outcomes in selected 
states, relying on administrative records gathered from several sources and quasi-experimental 
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TABLE 1. 1 



SERVICES MOST COMMONEY OEEERED BY 
TAEENT SEARCH PROJECTS IN 2000 



Type of Service 


Projects Providing Service 
(Percent) 


Academic Support 


Test taking and study skills 


98 


Academic advising and course selection 


94 


Career Development 


College orientation activities 


98 


Visits to college campuses 


96 


Referrals and counseling 


91 


Financial Aid 


Individual financial aid counseling 


98 


Einancial aid workshop 


97 


Assistance with pencil-and-paper EAESA 


97 


Scholarship searches 


94 



Source: Tables 6.1, 6.3, and 6.4 from Cahalan et al. 2004, based on MPR’s Survey of Talent Search 

Project Directors in 1999-2000. 



statistical techniques, which allow us to identify students who did not participate in the program 
but who were similar to Talent Search participants. 

In Chapter II, we describe the methods used in the analysis. In chapters III, IV, and V, we 
describe the data sources used to compile records on students, the strengths and weaknesses of 
the data, the specific analytic approach for identifying nonparticipating students who are similar 
to Talent Search participants in each state, and estimates of the effects of Talent Search on 
completion of high school, financial aid, and postsecondary enrollment in Texas, Indiana, and 
Florida.^ In the final chapter, we discuss our findings across the states and assess the success of 
our analytic approach. 

^ We present findings in Texas first since the data collected in Texas were compiled entirely by MPR staff. We 
follow with the findings in Indiana and Florida to provide contrast in the type of information available across the 
three states. The order of the presentation of the state findings do not reflect the importance of the findings in one 
state over another. 
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II. STUDY DESIGN AND METHODOLOGY 



A. STUDY DESIGN 

1. Research Questions 

The two main research questions posed in this design and addressed in this report are: (1) Is 
it possible to rely on administrative records to compile a complete, retrospective record of 
participation in Talent Search, characteristics of students in secondary school, and secondary and 
postsecondary outcomes? and (2) Can administrative data sources and quasi-experimental 
techniques be used to identify students who are not participating in the program but have 
characteristics similar to the Talent Search students in order to estimate the relationships between 
participation in Talent Search and secondary and postsecondary outcomes? 

2. State and Project Selection 

We chose Florida, Indiana, and Texas for our analysis based on the findings of the feasibility 
study (Maxfield et al. 2000). We also conducted additional research and negotiations in other 
states. The requirements for the administrative data were high: states had to provide secondary 
and postsecondary data that would allow us to track the experiences of one cohort of ninth-grade 
students over several years. In addition, we needed detailed identifying information on students 
so that we could merge the data from Talent Search projects and federal financial aid 
applications with the state data. A brief description of the data sources in each state is provided 
in the chapter on that state; Appendix B contains a more detailed explanation of the data sources 
and process of merging information across those sources. 

The cohort we chose for this study consisted of students in the ninth grade in the fall of the 
1995-96 school year. We chose this cohort to allow collection of retrospective data that enable 
us to assess outcomes related to the Talent Search program goals: high school completion. 
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application for financial aid, and college enrollment. Students from this cohort who graduated 



from high school “on time” in spring 1999 were first-time applicants for financial aid and new 
entrants in college in fall 1999. We obtained data on first-time applicants for financial aid and 
college enrollment for 1999, 2000, and (in some cases) 2001, thus allowing extra time for 
students who take longer to complete high school or wait after completing high school before 
applying for aid and enrolling in college. 

After we identified the states for the analysis, we attempted to collect student participation 
and identification data from all Talent Search projects in each state; thus, we tried to obtain 
information on a complete census of Talent Search participants from 1993-2000.^ We focused 
on 1993-2000 because the cohort chosen for the study, students who were in ninth grade in 
1995-96, progressed from middle school through high school during this time period. Many 
projects did not have records of participants as far back as 1993, but all projects that provided 
data for the study had records back to 1995. We received data from at least 60 percent of 
projects in Florida, Indiana, and Texas, but we did not receive data from all projects in any state. 
Therefore, our findings do not reflect the experience of the entire population of participants, even 
for the cohort of students in ninth grade in 1995-96, although they do reflect the experiences of 
most of the participants in the cohort in these three states. 



® Some Talent Search participants may have first participated in the program in years prior to 1995-96, when 
they were in middle school. In order to capture services received in middle school, we extended the window of 
Talent Search program participation back to 1993. 

^ Appendix B contains a detailed description of the data collected from Talent Search projects. 
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B. ANALYTIC APPROACH 



1. The Preferred Comparison Group 

Our goal in assessing the effect of Talent Search on participants was to determine what 
would have happened to Talent Search students if the program had not been available to them. A 
randomized study would have provided the most valid framework for such an analysis, splitting 
students who wished to participate in Talent Search into groups of those assigned to participate 
and those not assigned to participate. Because a randomized study was determined to be 
infeasible, however, we attempted to identify a group of students who were similar to Talent 
Search participants but who had not participated in the program. Within the quasi-experimental 
framework, the preferred set of comparison students would (1) be drawn from the same districts 
as the Talent Search participants (for each project) but not from the same high schools; (2) have 
persisted to the same point in high school; and (3) be similar on all observable characteristics. 

Drawing comparison students from other high schools is useful because students in target 
high schools who chose to participate in Talent Search when it was offered may be 
fundamentally different in important characteristics from those who chose not to participate. 
Some of these characteristics — such as motivation and aspirations — were not consistently 
available in the data sources we compiled. If Talent Search participants are fundamentally 
different from nonparticipants, the estimated effects of participation based on comparisons 
between these groups may, in part, be attributable to differences in motivation that predate 
students’ participation in Talent Search. If Talent Search staff targeted students with higher 
college aspirations than otherwise similar students, the analysis will overestimate the effects of 

o 

participation on outcomes. We believe that selecting students from the same districts as Talent 



* For information on the types of students Talent Search targeted, see Cahalan et al. 2004. 
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Search participants, but not from the same target high schools, may help to minimize bias from 
self-selection into Talent Search. 

The second characteristic of our preferred comparison group is that they persist to the same 
point in high school as Talent Search participants did before the program served them.^ Students 
who persisted through high school may differ from those who did not persist for reasons that are 
difficult to measure, such as motivation or a supportive home environment. Because we believe 
these unobservable characteristics to be important in determining educational outcomes, we 
sought to control for them by forcing an exact match on persistence through high school. The 
pool of potential comparison students shrinks as students drop out of high school or are unable to 
be tracked, but we are confident that the remaining pool of students is more likely to resemble 
Talent Search participants than all students who were in ninth grade in the fall of 1995. 

Finally, as with all matching analyses, we want to use a matching strategy to identify a 
comparison group that is as similar as possible to the participant group, particularly in observable 
characteristics correlated with educational outcomes, such as gender, economic status, and 
academic performance in high school. Therefore, comparison students should be as similar as 
possible to Talent Search participants on these characteristics, as well as on other related, 
observable characteristics. Without a similar distribution, or “balance,” on these key 
characteristics, it is difficult to ascertain whether differences in outcomes are due to participation 
in the program or to differences between the two samples. Furthermore, Rubin (2001) argues 
that regression analysis alone cannot adjust for substantial differences in the distribution of 
covariates between the two groups. Therefore, for the propensity score matching approach to be 



® Differences in persistence after Talent Search serves students could be attributable to Talent Search and 
should be included in the estimate of the effect of the program on participants. 
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statistically valid, participants and the comparison groups chosen through the matching 
procedure must have similar distributions on the observable characteristics. 

We defined the participant population as students in the ninth-grade cohort in 1995-96 
served by Talent Search projects between 1993 and 2000. The initial pool of potential 
comparison students was the population of all nonparticipants in the state who also were in the 
ninth grade in fall 1995 and for whom we were able to collect data. We stratified, by geography 
and year, samples of participants and nonparticipants, into several groups per state. 

2. Using Propensity Score Models to Identify the Comparison Group 

Within each group, we estimated propensity scores on which the two samples were to be 
matched (Rosenbaum and Rubin 1983 and 1985). An unweighted logit model was estimated to 
distinguish between members of the two samples; the dependent variable (participation in Talent 
Search) was coded as 1 for participants and 0 for nonparticipants. The explanatory variables 
varied by state; in each state, however, they included demographic, socioeconomic, and some 
academic characteristics. We used the model to estimate the log odds (the output of the 
propensity score model) of participating in Talent Search, conditional on the explanatory 
variables, for each participant and nonparticipant in the group. 

Within each group, we matched Talent Search participants to similar nonparticipating 
comparison students. Two students were deemed comparable when their log odds were 
sufficiently close. We matched each participant to all comparison students deemed 
comparable — a technique called “matching with replacement.” Talent Search participants for 

Participants were defined as students served between 1993 and 2000 by the Talent Search projects that 
provided data for the study. Some projects included information on the number of years students were served and 
the specific services students received, but this information was not consistently available. 

" The log odds of being in the participant sample equals the natural log of P(1 )/ P(0), where P(l) equals the 
probability of being in the participant sample, and P(0) equals the probability of being in the comparison sample. 
We predicted these probabilities using the estimated coefficients from the logit model. 
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whom we could not find a similar comparison student were excluded from the study because no 



comparison student could provide credible information about the outcomes we would have 
observed without the student’s participation in Talent Search. In each state, however, we were 
able to match 95 percent or more of Talent Search participants to at least one comparison 
student. All comparison students who were matched to one or more participant sample members 
were included in the matched comparison group. 

Another component of the algorithm for matching participant and comparison students was 
to select the appropriate “caliper range” that defines how close the log odds of a participant and a 
comparison student must be to be deemed a suitable match. Narrow caliper ranges can leave 
many participants unmatched when similar students might be found in the comparison sample; 
wide caliper ranges generate matches between students who may be less similar. We tested 
several caliper ranges in matching participants to the comparison students, based on fixed 
fractions of the standard deviation of the log odds within the matching group. Each caliper range 
generated different matched participant and comparison samples. Therefore, selecting a caliper 
range was equivalent to selecting a pair of matched participant and comparison samples. 
Ultimately, we selected a narrow caliper range which led to matched participant and comparison 
samples with a smaller number of significant differences in variables that describe baseline 
characteristics and academic achievement, while still finding matches for a high percentage 
of participants.^^ 



We used a caliper range of l/20th of the standard deviation of the log odds from the propensity score model, 
a small caliper to ensure that we matched only those who had propensity scores very close to the participant being 
matched. Due to the large size of the nonparticipant population in the administrative data available, we were able to 
obtain well-matched comparison groups using even smaller calipers. Using smaller calipers, however, led to more 
participants dropping out of the analysis because we could find no suitable comparison students. We were able to 
identify matches for 95 percent or more of all participants in each state. 
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As part of the matching procedure, students in the comparison sample who matched to 
participants were assigned weights based on their matches. The total of the weights assigned to 
the matched comparison sample is the same as the total number of Talent Search participants — 
that is, when using the weights, the participant and comparison samples are the same size. For 
all comparison group members who were within the caliper range for a participant, each 
comparison group member received an equal share of the participant’s weight. For example, if 
10 comparison students fell within the caliper range of participant A, each comparison student 
would receive a weight of 1/10 from participant A. Furthermore, because matching is done with 
replacement, a comparison group member could fall within the caliper range of several 
participants. If one of the comparison group members from the example above also matched 
participant B, who had five comparison students within the caliper range, that comparison group 
member would receive an additional weight of 1/5 from matching participant B. Each matched 
comparison student was assigned a weight equal to the sum of the weight contributions from 
each participant to whom he or she was matched. In the example above, the comparison student 
would receive a weight of 1/10 from participant A, plus 1/5 from participant B, for a total 
of 3/10. 

The tables in each chapter show the results of matching. In general, we were unable to draw 
comparison students with all three characteristics of the preferred comparison group described 
above: in Texas and Florida, we were able to control for persistence and match on observable 
characteristics but only within the target high schools. In Indiana, we matched well on 
observable characteristics within the target high schools but lacked data on persistence. 
Therefore, in each state we drew two different comparison groups to Talent Search participants, 
to test whether our findings were sensitive to the comparison group drawn. In each state our 
findings were consistent compared to the comparison group chosen. That is, the results were 
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similar regardless of the comparison group chosen. Therefore, in presenting our findings for 
each state in chapters III-V, we present results using only the comparison group drawn from 
students within the same target high schools who had the same balance of observable 
characteristics as Talent Search participants, allowing us to compare results across states. 

C. ESTIMATING THE EFFECT OF TALENT SEARCH ON SECONDARY AND 

POSTSECONDARY OUTCOMES 

I. Empirical Specification 

We used a regression-adjusted approach to estimate the effect of participation in Talent 
Search on high school completion, first-time application for federal financial aid, and college 
enrollment. To compute the average effect of participation in Talent Search, we estimate a 
statistical model that predicts the outcome of interest as a function of participation status and 
background characteristics (for example, sex, race, ethnicity, and academic-risk status). This 
approach allowed us to (1) adjust for the small remaining differences in observable 
characteristics between Talent Search participants and the matched comparison group and 
(2) increase the precision of our estimates. The basic form of the model is: 

= A + ’ 

where y, is the outcome of interest; P, equals 1 if the student participated in Talent Search and 
0 otherwise; A, is a vector that includes the student’s characteristics; is a random error term 
that captures the effects of unobserved factors that influence the outcome; and /3o, fii, and ^ are 
parameters or vectors of parameters to be estimated. The parameter of greatest interest is Pi, 
which shows the impact of participating in Talent Search on student outcomes. We estimate the 
parameters in the above equation using ordinary least squares for both categorical and 
continuous outcomes. All regression models are weighted to account for matching, as described 
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in the preceding section, and allow for potential heteroskedasticity of standard errors due to 
clustering at the project level. 

2. Standard Errors 

By using a caliper range to determine potential comparison students who are similar to any 
given participating student, rather than a one-to-one match, we increase the size of our matched 
comparison student sample. The weighting algorithm accounts for the number of matches per 
treatment student, and the standard error calculations from the outcomes equation estimation 

I ^ 

account for the variance in the weights. 

3. Reliability of Propensity Score Matching Methods 

In using a matched comparison group based on propensity score design, we adhered to the 
criteria developed by Rosenbaum and Rubin in a series of articles (Rosenbaum and Rubin 1983 
and 1985; Rubin 2001) for replicating a randomized design as well as possible. That is, we used 
as many covariates as possible in estimating our propensity models, we report findings only for 
matched comparison groups with a distribution of characteristics similar to those of Talent 
Search participants, and we test the sensitivity of our findings to alternative comparison groups 
that also meet these criteria. One study showed that using propensity score matching can 
replicate program evaluation findings based on experimental designs (Dehejia and Wahba 1999); 
however, other analyses of the same data did not support this finding (Smith and Todd 2004). In 
addition, other studies have found that (I) nonexperimental methods in general, and propensity 
score methods specifically, do not replicate findings from randomized studies well; and (2) there 



We estimated robust standard errors to account for clustering and used the “svyreg” procedure in Stata to 
account for the variance in the weights. 
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is no consistent pattern of nonexperimental studies over- or underestimating the impacts of 
programs found using randomized designs (Agodini and Dynarski 2004; Glazerman et al. 2003). 

The limitations in the matched comparison group design have implications for the findings 
presented in the following chapters. Because the data available and our strategy for drawing 
comparison groups differed across each state, we have the most confidence in findings that are 
consistent across states. We report project-level impacts to assist in the interpretation of our 
state-level findings, but we do not rely on findings on individual projects in our assessment of 
the overall success of Talent Search in the three states. 
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III. TEXAS 



A. INTRODUCTION 

In 1999, there were 17 Talent Search projects operating in Texas, serving more than 
15,000 students (Cahalan et al. 2004). Sixteen of these projects were operating in 1995. This is 
many more projects, serving many more students, than were operating in Florida or Indiana at 
the time.^^ The analysis in this chapter is based on more than 4,000 Talent Search participants in 
10 Talent Search projects throughout Texas who were in ninth grade in the fall of 1995, along 
with similar nonparticipants who were in the same school districts. The data we obtained from 
state agencies included secondary school records on the entire 1995-96 ninth-grade cohort and 
postsecondary school records on students in the cohort who had enrolled in public postsecondary 
institutions in Texas. Compared to the nonparticipants, the students participating in Talent 
Search had higher rates of high school completion, first-time application for federal financial aid, 
and enrollment in public postsecondary institutions in Texas. The magnitudes of these 
differences in rates varied, but the differences in application for financial aid were the largest. 

This chapter describes the data sources that were used to compile records on students. It 
also discusses the strengths and weaknesses of the data and the methods we used to identify 
nonparticipating students who were similar to Talent Search participants. We draw comparisons 
between Talent Search participants and nonparticipants to estimate the relationships between 
Talent Search participation and high school completion, application for financial aid, and 



While Texas served the largest number of students in absolute terms, Indiana served the highest proportion 
of the low-income secondary school students in the state, followed by Texas and Florida (see Table 3.21 in Cahalan 
et al. 2004). 
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postsecondary enrollment. The chapter concludes with a discussion of the findings and an 
assessment of the success of using these data to examine the effect of Talent Search in Texas. 



B. DATA 

We used four main data sources (Table III.l): 



1. Secondary School Records. The Texas Education Agency (TEA) provided these 
records, which included demographic, socioeconomic, and academic characteristics 
of the cohort of students in ninth grade in fall 1995. These data come from several 
TEA data collections. We obtained demographic and academic characteristics from 
the “fall snapshot” of ninth-graders in 1995. We took enrollment status, grade level, 
and high school exit status from data collected each summer following the school 
years from 1996 to 2000. We collected eighth-grade test scores, based on 
administration of the Texas Assessment of Academic Skills (TAAS) in spring 1994 
and spring 1995.'^ 

2. Talent Search Project Records. We obtained records from 13 of the 17 Talent 
Search projects operating in 1999; 11 projects provided information on all students 
served during the period of this study (1995-2000).'^ Although the various Talent 
Search projects provided different types of data, all projects provided the name, 
social security number (SSN), date of birth, and gender for the students they served. 
Project records also indicated the years a student was served. 

3. Federal Financial Aid Application Records. We obtained these records from the 
Office of Postsecondary Education (OPE) in ED. They contained the names of 
nearly all first-time applicants for federal financial aid who lived in Texas in 1999 
and 2000. 

4. Postsecondary School Records. The Texas Higher Education Coordinating Board 
(THECB) supplied these records. They included information on enrollment, part- 
time or full-time enrollment status, credits earned, and enrollment in a two- or four- 
year institution for all students in the cohort who enrolled in a public college or 
university in Texas for the school years 1999-2002. 



As discussed in Chapter II, we selected the cohort in this study to allow the collection of 
retrospective data in order to assess several outcomes related to the goals of the Talent Search 



In 2003, the Texas Assessment of Knowledge and Skills (TAKS) replaced the TAAS. 

One additional project began operating in 1998-99 and did not serve enough students in the target cohort to 
be included in the analysis, thus all analyses are based on participants served by 10 projects. 
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TABLE III.l 



TEXAS DATA SOURCES 



Source 


Variables 


Years 


Number of Students from 
Pall 1995 Ninth-Grade 
Cohort 


Texas Education Agency 
(TEA) Public Education 
Information Management 
System (PEIMS) 


Name, SSN, and date of 
birth for all students in the 
ninth grade 


School year 1995-96 


335,563 


Texas Education Agency 
(TEA) Public Education 
Information Management 
System (PEIMS) 


Enrollment status, grade 
level, and high school exit 
status 


Summer 1996 
Summer 1997 
Summer 1998 
Summer 1999 


329,019'’ 

297,020 

265,759 

218,372 


Texas Education Agency 
Student Assessment 
Division 


Eighth-grade scores on the 
Texas Assessment of 
Academic Skills (TAAS) 


Spring 1994 
Spring 1995 


273,866 


U.S. Department of 
Education, Office of 
Postsecondary Education 
(OPE) 


Eirst-time applicants for 
federal financial aid who 
were living in Texas 


School year 1999-2000 
School year 2000-01 


86,145*’ 


Texas Higher Education 
Coordinating Board 
(THECB) 


Postsecondary enrollment, 
credits, and type of 
institution 


School year 1999-2000 
School year 2000-01 
School year 2001-02 


122,055 


Talent Search Projects 


Lists of students and years 
served 


1993-2000 


4,177” 



Note: Figures indicate the number of students in the ninth-grade cohort who were identified with records in the 

other data sources. 

“The number of cohort students remaining declines each year as students leave Texas public schools. 

’’There were approximately 180,000 first-time applicants for federal aid from Texas in 1999-2000 and 2000-01. 
“Based on a total of 31,000 students served by 10 Talent Search projects in the years 1993-2000. 
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program, including high school completion, application for financial aid, and college enrollment. 
Students in this cohort who graduated from high school “on time” in spring 1999 were first-time 
applicants for financial aid and new entrants in college in fall 1999. We obtained data on first- 
time applicants for financial aid for both 1999 and 2000, to expand the window for students who 
take an additional year to complete high school or wait a year after completing high school 
before applying for aid. Similarly, we collected data on postsecondary enrollment for 1999, 
2000, and 2001, to allow students two extra years to complete high school and enroll in college 
and to assess initial persistence in college for as many in the cohort as possible. Altogether, 
these data sources yielded records on 335,563 students who were in ninth grade in a Texas public 
school in fall 1995.^^ Talent Search projects that provided data served 4,177 of these students at 
some point during secondary school. 

1. Strengths and Weaknesses of the Data 

The use of administrative records resulted in a database with several strengths for studying 
the effect of Talent Search on participants: 



• The data are comprehensive and comprise the full sample of ninth-graders in 
Texas public schools in the fall of 1995. Because Talent Search projects are spread 
throughout the state, a critical requirement was to be able to draw comparisons for 
students from across the state. MPR staff took care to merge the records from several 
data sources using procedures that were thorough and accurate. 

• The data contain important predictors of high school completion, application for 
financial aid, and college enrollment. These characteristics include race, ethnicity, 
gender, economic status (defined as eligibility for free or reduced-price lunches), a 
state indicator for being at-risk of dropping out of school, high school curriculum, 
participation in gifted and talented programs, and test scores. Identifying comparison 
students using statistical modeling is more feasible when we can determine whether 
students are similar in characteristics related to the outcomes studied. 



We restricted analysis to students in the ninth grade who were between 13.7 and 17 years of age in fall 1995. 
This reduced the analysis sample to 321,000. 
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• The data include persistence through high school, which allows us to control for 
important observable and related unobservable characteristics in selecting 
comparison students. It is essential that the many Talent Search students who are not 
served until they are juniors or seniors in high school be compared only to students 
who have also progressed to a similar point in school. 

• The data provide reliable information on key outcomes, such as high school 
completion status, application for federal financial aid, enrollment in college, and 
initial persistence in college. These outcomes, because they are based on data that 
come from administrative sources, are likely to be more accurate and comprehensive 
than results based on self-reported information. 



The data also have limitations for determining differences between Talent Search 
participants and similar nonparticipants: 



• The data do not contain some characteristics that are important predictors of key 
outcomes. These include the parents’ education level, family structure, and education 
and career aspirations and plans. It would be useful to control for these variables in 
the analysis, especially if Talent Search staff target students who already aspire to 
attend college; without them, we are likely to overstate the effect of participation in 
Talent Search. 

• Students drop out of the data as the state loses track of them. Although grade level 
and high school completion status are recorded for each student in each year, the state 
loses track of some students. In the target cohort in this study, Texas did not record 
an exit status for 22 percent of the sample. Many of those students are also missing 
grade level enrollment information at some point. Therefore, the sample suffers from 
attrition even though initially it was a complete census of the cohort. 

• Postsecondary enrollment data are limited to attendance at public colleges and 
universities in Texas. This limitation implies that overall postsecondary enrollment 
rates of students in this cohort are understated. However, this affects our estimates of 
the difference in postsecondary enrollment between Talent Search participants and 
comparison students only if one group is more likely to enroll in private or out-of- 
state institutions. 

• Information on participation in Talent Search, application for federal financial 
aid, and postsecondary enrollment is not integrated with information on secondary 
schooling. The data that Talent Search projects, OPE, and THECB provided do not 
share a common identification system with the TEA. Although all data sources 
included SSNs for students, 10 percent of students in the TEA data have no valid 
SSNs. Some of these cases could be matched to SSNs using names and dates of 
birth; however, some of the students in the cohort who participated in Talent Search, 
applied for financial aid, or enrolled in college probably remain unidentified. 
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• We did not gather data for all Talent Search participants in the ninth-grade cohort, 
as some projects did not supply data. If projects not supplying data are 
systematically different from those that did provide data, the study findings will not 
apply to all the students served by Talent Search in Texas. 

The compilation of the data in Texas demonstrates the feasibility of extracting information 
from administrative records to study the effect of Talent Search on participants. The 
comprehensiveness and accuracy of the data are the greatest advantages of this approach. On the 
other hand, given that educational aspirations are an important factor affecting the outcomes of 
Talent Search participants, the lack of information on aspirations is a serious limitation of 
the data. 

2. Characteristics of Talent Search Participants and Potential Comparison Students 

The characteristics of Talent Search participants differed significantly from those of 
nonparticipants across the state as a whole, as well as within the high schools that individual 
projects targeted. We identified target high schools through the projects’ 1999 Annual 
Performance Reports (APRs) and verified them through MPR’s survey of projects in 1999-2000. 
Table III. 2 shows that Talent Search students were more likely than all other students in the state 
to be black or Hispanic, economically disadvantaged, and taking a vocational or technical 
course — all characteristics associated with lower rates of high school completion and college 
enrollment. On the other hand, they also had characteristics associated with higher rates of 
high school completion and college enrollment: they were more likely than nonparticipants to be 
female, to be the traditional age for entering ninth grade, and to be participating in a gifted and 
talented program. They also had slightly higher eighth-grade test scores. 



** Appendix Table A.III.l defines the variables used in the analysis. 
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